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SUMMARY 


We  developed  several  novel  representational  and  processing  techniques  for  use  in  connectionist 
systems  designed  for  high-level  Al-like  applications  such  as  commonsense  reasoning  and  natural 
language  understanding. 

The  techniques  were  used,  lor  instance,  in  a  connectionist  system  (Conposit/SYLL)  that 
implements  Johnson- Laird’s  mental-model  theory  of  human  syllogistic  reasoning.  This  theory  was 
chosen  as  a  case  study  for  verifying  the  power  of  the  techniques,  because  it  was  developed  indepen¬ 
dently  of  the  project,  contains  complex  symbolic  structures  of  various  types,  and  requires  complex 
sequences  of  operations.  The  resulting  connectionist  system  is  probably  the  most  advanced,  com¬ 
plex,  and  complete  connectionist  rule-based  system  in  existence.  It  has  a  more  complete  scheme 
for  binding  rule-variables  and  for  role  binding  than  any  other  connectionist  system. 

The  representational  techniques  developed  in  the  project  were  Relative- Position  Encoding 
(RPE)  and  Pattern- Similarity  Association  (PSA).  RPE  allows  structure  to  be  encoded  by  the 
relative  positioning  of  connectionist  activation  patterns  within  a  subnetwork.  PSA  allows  structure 
to  be  encoded  by  having  different  substructures  include  similar  activation  subpatterns.  These 
techniques  are  similar  to  data  structuring  techniques  used  in  computer  memory  —  in  particular, 
PSA  is  similar  to  associative  addressing  —  but  they  had  not  previously  been  used  in  any  non-trivial 
way  in  connectionism. 

The  most  distinctive  processing  technique  developed  in  the  project  was  the  Temporal- 
Winner-Take- All  (TWTA)  method  for  selection  in  connectionist  networks.  This  is  more  convenient 
and  efficient  for  some  purposes  than  conventional  WTA  methods.  It  works  on  the  basis  of  signal¬ 
timing  differences,  as  opposed  to  the  activation-level  differences  used  in  conventional  WTA. 

There  were  three  other  significant  accomplishments  in  the  project.  First,  we  designed  and 
started  the  implementation  of  a  system  (ABR-Conposit)  that  performs  case /analogy-based  reason¬ 
ing  using  the  above  techniques  aud  others.  This  system  preserves  the  complex  structure  processing 
power  of  symbolic  systems  while  also  attaining  advantages  normall}  associated  with  connection¬ 
ism.  such  as  graceful  degradation.  ABR-Conposit  includes  a  type  of  reduced  representation  to  aid 
in  structure  matching  an  long-term  memory  retrieval. 

Secondly,  we  have  commenced  a  unique  study  into  the  ways  in  which  a  connectionist  system 
might  represent  and  reason  about  the  beliefs  and  other  mental  states  of  argents  with  which  it  is 


interacting.  Belief  representation/ reasoning  is  a  crucial  topic  within  the  study  of  commonsense 
reasoning  and  natural  language  understanding.  We  have  looked  in  particular  at  the  difficulties  in 
doing  belie!  representation/reasoning  in  connectionist  systems  that  perform  holistic  processing  of 
learned  reduced  representations. 

Thirdly,  we  have  identified  the  problems  raised  by  variables  within  working  memory  items 
(as  opposed  to  variables  in  processing  rules,  which  are  the  usual  sort  of  variables  commented  on  in 
connectionism).  We  have  shown  how  logical  combinulors  could  be  used  to  overcome  the  problems 
to  a  large  extent. 

The  major  tasks  set  out  in  the  Statement  of  Work  in  the  grant  proposal  were  accomplished, 
except  that: 

(a)  the  work  on  interpreted  rules  was  replaced  by  the  more  general  and  powerful  notion  of 
analogy/ case- based  reasoning; 

(b)  the  work  on  spatial  reasoning  was  set  aside  in  lavor  o!  the  work  on  analogy /case-based  reasoning 
and  on  belief  representation  and  reasoning. 


I 


EXPLANATIONS  OP  MAJOR  ACCOMPLISHMENTS 


Most  of  th<*  accomplishments  of  the  project  have  been  reported  in  refereed  journal  articles,  book 
chapters,  refereed  conference  papers,  technical  reports,  and  a  Ph.D.  thesis.  The  following  is  there¬ 
fore  confined  to  giving  brief  explanations  and  pointers  to  the  relevant  articles.  The  articles  are 
listed  in  later  sections. 

Relative-Position  Encoding  and  Pattern-Similarity  Association 

These  techniques  are  for  realizing  complex  data  structures,  which  might  for  instance  be  akin  to 
semantic  network  fragments,  in  working  memories  in  connectionist  networks,  in  such  a  way  as  to 
support  dynamic  infereucing  (i.e.,  the  types  of  inferenciug  needed  in  commonsense  reasoning  and 
natural  language  understanding).  Relative- Position  Encoding  ( RPE)  achieves  dynamic  associations 
between  data  structure  components  in  working  memory  by  putting  the  connectionist  activation 
patterns  encoding  those  components  in  suitable  relative  positions.  The  particular  form  of  RPE  we 
have  studied  most  uses  an  array-structured  connectionist  subnetwork  as  the  working  memory  of 
our  C’onposit  system.  The  data  structure  components  take  the  form  of  activation  patterns  that 
can  appear  anywhere  in  the  array. 

Degenerate,  simplistic  forms  of  RPE  have  appeared  in  many  connectionist  systems,  no¬ 
tably  in  vision  systems  and  system  that  have  input  or  output  layers  divided  into  separate  buffers. 
However,  the  full  power  of  the  technique  had  not  previously  been  exploited.  The  power  results 
from  the  convenience  of  being  able  to  mutually  structure  data  items  by  merely  positioning  them 
in  dynamically  chosen  places,  rather  than  by  the  more  usual  connectionist  means  of  putting  them 
in  pre-assigned  places  or  facilitating  connection  pathways  (by  changing  weights  or  setting  binding 
node  activations).  RPE  is  a  generalization  of  the  basic  computer  data  structuring  technique  of 
placiug  related  data  fields  in  contiguous  positions  in  main  memory. 

The  only  other  connectionist  research  we  are  aware  of  that  uses  anything  like  a  non-trivial 
form  of  RPE  is  the  work  on  natural  language  parsing  by  Charniak  and  Santos  and  work  on  a 
connectionist  Lisp-like  system  by  Goebel. 

Pattern-Similarity  Association  ( PSA )  achieves  dynamic  associations  between  data  structure 
components  in  working  memory  by  including  in  the  components  activation  sub-patterns  that  act 
as  associative  keys.  The  keys  are  dynamically  chosen  or  created,  and  have  no  pre-defined  meaning. 


Two  components  that  contain  the  same  key  or  sufficiently  similar  keys  are  deemed  to  be  associated 
with  each  other.  This  technique  is  a  direct  extrapolation  from  associative  addressing  in  computers. 

Associative  addressing  is  at  least  as  powerful  as  the  pointer  method  for  data  structuring, 
and  is  a  much  more  natural  candidate  for  transfer  into  connectionism  than  pointers  are.  It  is 
somewhat  remarkable,  therefore,  that  previous  counectionist  research  has  not  adopted  the  idea. 
The  only  other  connectionists  research  we  know  of  that  uses  a  form  of  PSA  is  the  work  of  Shastri 
and  Ajjanagadde  on  role  binding  by  means  of  time-phase  sharing.  The  time  phases  can  be  viewed 
as  associative  keys. 

The  type  of  PSA  we  have  focused  on  is  used  in  the  array-structured  working  memories 
mentioned  above,  in  a  tight  combination  with  RPE. 

Both  techniques  are  presented / discussed  in  Bartiden  (1988a,  1989a,  1991),  Barnden  and 
Srinivas  (1991,  1992a)  and  elsewhere.  In  particular,  Barnden  and  Srinivas  (1991),  article  in  the 
journal  Connection  Science,  extensively  discusses  the  relationships  of  the  techniques  to  other  con- 
nectionist  information-structuring  methods,  and  also  to  methods  used  in  computers.  The  rich 
inter- relationsliips  of  these  other  counectiomst  and  computer  methods  are  also  clarified. 

Temporal- Winner- Take- All 

Winner-Take- All  (WTA)  networks  frequently  appear  in  neural  network  models.  They  are  pri¬ 
marily  used  for  decision  making  and  selection.  As  an  alternative  to  the  conventional  activation 
based  Winner- Take- AH  mechanisms  (AWTA),  we  present  a  fime-6ased  Temporal- Winner- Take-All 
(TWTA)  mechanism  with  O(n)  space  complexity  and  roughly  0(logn)  time  complexity.  The  mech¬ 
anism  exploits  systematic  and  stochastic  differences  between  time  delays  within  different  units  and 
connections.  The  TWTA  and  the  AWTA  networks  are  shown  to  be  logically  equivalent,  but  the 
TWTA  mechanism  is  more  suitable  than  the  latter  for  various  selection  tasks,  especially  the  selec¬ 
tion  of  an  arbitrary  unit  from  a  set  (e.g.  as  in  unit  recruitment).  TWTA  avoids  various  problems 
with  conventional  WTA,  notably  the  difficulty  of  making  it  converge  rapidly  over  a  large  range 
of  conditions.  We  have  performed  a  probabilistic  analysis  of  the  TWTA  mechanism  and  obtained 
experimental  data  from  numerous  massively  simulations  of  the  TWTA  mechanism  on  a  Connection 
Machine. 

TWTA  is  used  for  various  selection  purposes  in  the  Conposit  system. 


(i 


The  method  is  presented  and  discussed  in  Barnden  (1991),  Baxnden  and  Srinivas  (in  press), 
Barndeu,  Srinivas.  k  Dharmavaratha  (1990),  Srinivas  k  Barnden  (1990),  and  Srinivas’s  Ph.D. 
dissertation. 


Rule-Based  Reasoning  System  (Conposit) 

Conposit  is  a  structured  (but  not  classically  localist)  connectionist  rule-based  system.  Its  rules 
operate  on  an  array-structured  working  memory  of  the  sort  mentioned  above,  using  the  Relative- 
Position  Encoding  and  Pattern-Similarity  Association  data  structuring  techniques.  The  action  part 
of  each  rule  is  implemented  as  a  flowchart-like  connectionist  subnetwork  that  changes  the  contents 
of  working  memory  by  sending  streams  of  signals  to  it.  The  action  part  is  triggered  by  the  output 
of  a  subnetwork  that  can  detect  the  presence  of  specific  data  structure  fragments  in  the  working 
memory.  This  detection  module  is  highly  parallel  and  efficient.  Temporal-Winner-Take- All  is  used 
in  Conposit  for  rule  selection,  random  decisions  to  exit  from  loops  within  action  parts,  selection  of 
data  structure  fragments  to  work  on  within  the  working  memory,  and  recruitment  of  free  space  in 
working  memory  for  new  data  structure  fragments. 

The  main  version  of  the  Conposit  system  is  Conposit/SYLL,  which  implements  syllogistic 
reasoning  through  Johnson-Lnird  mental  models,  though  with  major  modifications  to  the  way 
they  are  processed.  There  was  no  immediate  intention  to  extend  the  Johnson-Laird  approach  as  a 
psychological  theory.  Rather,  the  exercise  was  undertaken  in  order  to  test  the  power  of  the  RPE, 
PSA  and  TWTA  techniques. 

The  application  of  these  techniques  to  mental- model  manipulation  raises  some  broad  issues. 
One  is  the  usefulness  of  marker- passing  over  dynamically-arising,  virtual  networks  arising  in  the 
activation  state  of  working  memory  at  a  high  level  of  description,  rather  than  over  static  networks 
at  the  basic  architectural  level  of  the  system.  Another  is  tiie  way  in  which  the  particular  nature  of 
the  techniques  affects  what  it  is  natural  to  do  at  the  symbolic  level  of  description.  In  particular, 
the  techniques  predispose  the  system  towards  random  as  opposed  to  pre-ordered  sequencing  of 
subtasks,  and  towards  associative  linking  as  opposed  to  use  of  explicit  linking  constructs. 

Conposit  is  elaborate,  and  is  geared  towards  procedurely  more  elaborate  reasoning  tasks 
than  those  tackled  by  other  connectionist  reasoning  systems.  It  is  described  and  discussed  primarily 
in  Barnden  (1989a,  1990.  1991).  -See  also  Barnden  and  Srinivas  (1991). 


Conuectionist  Analogy /Case- Bused  Reasoning 

This  work  addressed  the  question  of  how  we  are  to  gain  both  the  standard  advantages  of  connec- 
tionistu  and  those  of  symbolic  systems,  without  adopting  hybrid  symbolic/connectionist  systems. 
Fully  conuectionist  systems  that  support  analogy- based  reasoning  are  proposed  as  an  answer,  at 
least  in  the  realm  of  high-level  cognitive  processing.  This  domain  includes  commonsense  reason¬ 
ing  and  the  semantic/pragmatic  aspects  of  natural  language  processing.  The  proposed  type  of 
system,  purely  by  being  analogy- based,  gains  forms  of  graceful  degradation,  representation  com¬ 
pletion,  similarity-based  generalization,  learning,  rule-emergence  and  exception-emergence.  The 
system  therefore  gains  advantages  commonly  associated  with  connectionism,  although  the  precise 
forms  of  the  benefits  are  different.  At  the  same  time,  through  being  fully  connectionist,  the  system 
also  gains  the  traditional  connectionist  variants  of  those  advantages,  as  well  as  gaining  further 
advantages  not  provided  by  analogy- based  reasoning  per  se.  And,  because  the  system  is  in  part 
an  implementation  of  a  form  of  symbolic  processing,  it  preserves  the  flexible  handling  of  complex, 
temporary  structures  that  are  well  supported  in  traditional  artificial  intelligence  and  which  are 
essential  for  high-level  cognitive  processing.  The  work  is  in  part  a  reaction  against  the  excessive 
polarization  of  the  connectiotusm/symbolicism  debate.  This  polarization  is  seen  as  resulting  from 
over-simplified,  monolithic  views  both  of  what  symbolic  processing  encompasses  and  of  the  nature 
of  the  benefits  that  connectionism  provides. 

The  system  is  a  new  version  of  Conposit,  called  ABR-Conposit.  It  has  been  partially 
implemented.  A  preliminary  design  is  reported  in  Barnden  and  Srinivas  (1992a).  See  also  Barnden 
(forthcoming,  a,  b). 

ABR-Conposit  replaces  the  arbitrary  “associative  keys”  used  for  PSA  in  the  older  Conposit 
versions  by  associative  keys  that  are  dynamically  computed  from  their  local  contexts  in  the  working 
memory.  These  keys  are  a  form  of  (uon-learned)  reduced  representation. 

Belief  Representation/Reasoning 

The  systemalicit y  and  structure-sensitivity  of  high-level  cognition,  notably  commonsense  reasoning 
and  natural  language  understanding,  are  widely  recognized  to  present  a  challenge  to  connectionism. 
However,  an  aspect  of  systematicity  and  structure-sensitivity  has  not  been  adequately  addressed 
by  either  side  of  the  symbolicist/connectionist  debate.  A  system  must  have  a  way  of  embedding 
reasoning  within  various  types  of  context.  For  instance,  a  system  must  be  able  to  reason  within  the 


contest  of  another  agent's  beliefs.  This  requirement  present  particular  challenges  to  connectionist 
systems  t hat  work  by  learned  holistic  processing  of  compi  ssed  encodings  (reduced  representa¬ 
tions).  Such  systems  have  been  shown  to  have  considerable  promise  for  providing  systematicity 
and  structure-sensitivity  while  avoiding  a  straightforward  implementation  of  traditional  symbol 
manipulation.  However,  there  are  reasons  to  think  that  such  systems  will  have  great  difficulty  do¬ 
ing  belief  represen tation/reasouing  holistically,  that  is  without  breaking  structures  down  into  parts 
as  is  done  in  symbolic  systems.  A  preliminary  simulation  study  by  Balogh,  one  of  the  students 
supported  by  the  grant,  has  provided  preliminary  verification  of  this  contention,  but  the  necessary 
experimental  program  is  large  and  will  be  very  time-consuming. 

The  topic  of  belief  representation/reasoning  has  barely  been  touched  on  in  other  connec¬ 
tionist  work.  It  is  to  be  carefully  distinguished  from  quasi-connectionist  work  that  has  been  done 
misleadingly  under  the  heading  of  "belief.”  in  which  wliat  is  at  issue  is  merely  the  “belief”  set  of 
the  system  itself. 

Our  work  on  connectionism  and  beliefs  is  reported  in  Barnden  (1992a,  b  c,  d). 

Avoiding  Variables  by  means  of  Logical  Combinators 

The  connectionist  problem  of  achieving  the  quanlificational  effect  of  symbolic  variables  is  well 
recognized.  However,  one  relatively  neglected  issue  is  that  of  variables  in  working  memory  repre¬ 
sentations  (arising,  for  instance,  from  natural  language  inputs),  as  opposed  to  variables  in  rules. 
Working  memory  variables  present  difficulties,  centering  on  the  arbitrariness  of  the  set  of  variables 
used  in  any  given  expression,  and  on  non-uniformity  in  expressions  and  manipulations.  However, 
the  variables  can  in  principle  be  avoided,  for  instance  by  using  logical  combinators.  These  are 
special  functions  much  studied  within  the  symbol  processing  arena.  The  use  of  combinators  makes 
structures  less  arbitrary  and  more  uniform.  The  reduced  arbitrariness  ameliorates  an  important 
systematicity  problem,  and  the  added  uniformity  could  facilitate  high-level  parallelism.  We  do  not 
claim  i  hat  the  combinator  approach  is  definitely  the  right  one  to  adopt,  because  of  some  problems. 
Nevertheless,  combinators  need  to  be  borne  in  mind,  and  the  synibolic/connectionist  debate  has 
been  over-simplified  in  ignoring  them.  We  have  investigated  severed  ways  in  which  combinator- 
based  working  memory  items  could  be  implemented  in  connectionism,  with  special  attention  to 
reduced- representation  implementations.  (Also,  combinators  could  easily  be  implemented  in  Con- 
posit.) 
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This  work  lias  so  Tar  been  largely  theoretical,  and  is  discussed  briefly  in  Barnden  and 
Srinivas  (1990)  and  extensively  in  Barnden  and  Srinivas  (1992b). 
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“Encoding  Complex  Symbolic  Data  Structures  with  Some  Unusual  Connectionist  Techniques.” 
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(I  was  co-organizer  of  the  workshop,  which  was  attended  by  prominent  researchers  from 
across  the  country.) 

“The  Right  of  Free  Association:  Relative- Position  Encoding  for  Connectionist  Data  Structures.” 
Poster  paper  at  lUtli  Annual  Con  fere  net  of  the  Cognitive  Science  Society,  Montreal,  August 

ms. 

“Commousense  Reasoning  in  Conposit,  a  Quasi-Connectionist  Register- Array  Model.” 
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Oral  presentation  at  Annual  Meeting  of  the  Society  for  Philosophy  and  Psychology,  Tucson, 
April  1989. 
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works.” 

Poster  presentation  at  1st  International  Joint  Conference  on  Neural  Networks,  Washington, 
D.C.,  June  1989. 

“Neural- Net  Implementation  of  Complex  Symbol-Processing  in  a  Mental  Model  Approach  to  Syl¬ 
logistic  Reasoning.” 
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"Handling  of  Noise,  Irrelevance  and  Inferential  Conflict  in  a  Connectionistically  Implemented  Case- 
Based  Reasoning  System.” 
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Presentation  of  discussion  points  while  acting  as  moderator  of  a  session  at  Workshop  on  Con¬ 
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sentation  and  Inference.) 

“Connectionism  and  Propositional  Attitudes.” 
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“Secrecy,  Separation,  Singleness:  The  Modules  of  Modularity.” 
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Models  of  Cognition.  5th  Annual  Conference  on  .Neural  Information  Processing  Systems, 
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“Beliefs,  Connectionism,  Meta- Representation,  Vagueness:  Stirring  the  Pot.” 

Oral  presentation  at  Symposium  on  Propositional  Knowledge  Representation ,  AAAI  Spring 
Symposium  Series.  Stanford  University.  March  1992. 
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"Belief  representation  and  connectionism." 

Oral  presentation  at  AAAI-9'2  Workshop  on  Integrating  Neural  and  Symbolic  Processes — 
The  Cognitive  Dimension.  San  Jose.  CA.  July  1992. 


Regional/Local 

"Simulations  of  a  cognitive  model  on  the  Massively  Parallel  Processor.” 

Oral  presentation  in  Computer  Science  Department  colloquium  series,  NMSU,  September 
1987. 

"High-Level  Reasoning  in  a  Quasi-Connectionist  Register- Array  Model.” 

Oral  presentation  at  3rd  New  Mexico  Computer  Science  Conference ,  UNM,  Albuquerque, 
April  1988. 

"The  Right  of  Free  Association:  Encoding  Techniques  for  Connectionist  Data  Structures.” 

Oral  presentation  in  ACM  Colloquium  Series,  Computer  Science  Department,  NMSU,  De¬ 
cember  1988. 

"From  High-Level  Cognition  to  Neural  Networks”. 

Invited  oral  presentation  at  the  session  on  Neural  Computing  Networks  at  Regional  Meeting 
of  the  American  Association  for  the  Advancement  of  Science ,  Las  Cruces,  April  1989. 

"Temporal-Winner-Take-All  Networks  for  Arbitrary  Selection  in  Connectionist  and  Neural  Net¬ 
works.” 

Oral  presentation  at  4th  New  Mexico  Computer  Conference,  NMIMT,  Socorro,  April  89. 

"Conposit.  a  Neural  Network  System  lor  Symbolic  Processing.” 

Invited  oral  presentation  at  meeting  of  the  Rio  Grande  Chapter  of  the  Association  for 
Computing  Machinery.  UNM.  Albuquerque.  September  1989. 

"High- 1  .evol  Cognitive  Processing  in  Connectionist/Neural  Networks.” 

Invited  oral  presentation  to  Statistics  Group,  Mathematics  Department,  NMSU,  April  1990. 


"Connections  by  Analogy  to  Symbols." 

Colloquium.  Computer  Science  Department,  New  Mexico  State  University,  November  1991. 


Presentations  by  Co-Authors 

The  following  presentations  were  given  by  grant-supported  students  who  co-authored  the  papers 
with  me. 

“Cross- Fertilization  between  Connectionist  Networks  and  Highly  Parallel  Architectures.” 

Oral  presentation  by  K.  Srinivas  at  5th  IASTED  International  Conference  on  Expert  Sys¬ 
tems  and  ,\ eii rul  Networks,  Honolulu.  August  1989.  (IASTED  =  International  Association 
of  Science  and  Technology  for  Development.) 

“Winner-Take- AJ1  Networks:  Time- Based  versus  Activation-Based  Mechanisms  for  Various  Selec¬ 
tion  Coals." 

Oral  presentation  by  lv.  Srinivas  at  IEEE  International  Symposium  on  Circuits  and  Sys¬ 
tems,  New  Orleans,  May  1990. 

“Competition  and  Selection  in  Neural  Networks  with  Distributed  Representations.” 

Or«d  presentation  by  I\.  Srinivas  at  Rocky  Mountain  Conference  on  Artificial  Intelligence, 
N.MSL’,  June  1990. 

“Applications  of  Connectionism  to  Analogical  Reasoning.” 

Oral  presentation  by  T.C.  Eskridge  at  Jrd  Midwest  Artificial  Intelligence  and  Cognitive 
Science  Conference.  Southern  Illinois  University  at  Carbondale,  April  1991. 
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Balogh.  I.,  Ph.D.  Student,  Computer  Science  Department,  New  Mexico  State  University. 
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